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Abstract. Let f2 be a vector space over a finite field with q elements. Let 
G denote the general linear group of endomorphisms of Q and let us consider 
the left regular representation p : G — > £?(Z/2(X)) associated to the natural 
action of G on the set X of linear subspaces of f2. In this paper we study a 
natural basis B of the algebra EndQ(L2(X)) of intertwining maps on L2QC). 
By using a Laplacian operator on Grassmann graphs, we identify the kernels 
in B as solutions of a basic hypergeometric difference equation. This provides 
two expressions for these kernels. One in terms of the g-Hahn polynomials and 
the other by means of a Rodrigues type formula. Finally, we obtain a useful 
product formula for the mappings in B. We give two different proofs. One uses 
the theory of classical hypergeometric polynomials and the other is supported 
by a characterization of spherical functions in finite symmetric spaces. Both 
proofs require the use of certain associated Radon transforms. 



Introduction 

Let X be a homogeneous space with respect to a given finite group G. That is, 
the group G acts transitively on the set X. Then, we can consider the left regular 
representation p : G — > B(Vx) of G into the Hilbert space Vx of complex-valued 
functions if : X — >• C. In this context, it is well-known that the algebra Endc(Vx) 
of intertwining operators with respect to p codify some relevant information. For 
instance, Endc(Vx) is abelian if and only if the left regular representation p is 
multiplicity- free. In this case, following Terras' book we say that X is a 

finite symmetric space with respect to G. When dealing with finite symmetric 
spaces, any explicit expression for the kernels of the orthogonal projections onto 
the irreducible components of Vx is interesting. Indeed, the main motivation lies 
in the fact that these expressions can be usually regarded as combinatorial versions 
of the irreducible characters. More generally, if X and Y are finite symmetric 
spaces with respect to G, some information about the relations between X and Y 
in the group theory level can be obtained by studying the space Hohig(Vx, Vy) of 
intertwining homomorphisms from Vx to Vy- For instance, the dimension of the 
space HoniG(Vx,VY) coincides with the number of irreducible components that 
Vx and Vy have in common. 

Two natural problems arise in this setting. In order to state them, let us consider 
the Radon transform 1Z Z : Vx Vy associated to a G-invariant subset ZcXxY 
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Taking the G- invariant subset Z to be each of the orbits Oi, O2, ■ ■ ■ , Od of the 
action of G on the product X x Y, we obtain a basis of the space HoniG(Vx, Vy) 
made up of Radon transforms 

Bi = \ni,n 2 , . . . ,n d y 

On the other hand, since the dimension of Homc(Vx, Vy) coincides of the number 
of irreducible components that Vx and Vy have in common, we obtain the following 
orthogonal decompositions 

V x = Wxffi V x ,«, 

l<s<d 

Vy = Wy© Vy,., 
l<s<d 

where Vx.si is equivalent to Vy,s 2 if and only if si = s 2 . Then we introduce 
non-zero operators A s S Hohig(Vx,s, Vy. s ) and we regard them as elements of 
Houlg(Vx,Vy) vanishing on Vx 8 Vx jS . Each mapping A s is an intertwining 
isomorphism between Vx,s and Vy,s and, by Schur lemma, it is unique up to a 
constant factor. This family of mappings provide another basis of HomG(Vx, Vy) 

B 2 = {Ai,A 2 ,...,A d }. 

Both bases Bi and B2 are orthogonal with respect to the Hilbert-Schmidt inner 
product. The first problem we are interested on is to obtain the coefficients relating 
the bases described above. On the other hand, given three finite symmetric spaces 
Xi,X2 and X3, operator composition provides a bilinear mapping 

Hom G (V X2 , Vx 3 ) x Hom G (V Xl , V X2 ) — » Hom G (V Xl , Vx 3 ) 

given by 

( A 2,3 )A l, 2) ^ A 2,3 oA l,2_ 

The second problem we want to study is to write the products A 2 ' 3 o A*' 2 in terms 
of the operators A*' 3 . In this paper we solve the problems presented above for the 
Grassmann graphs associated to the general linear group over a finite field. For 
the first problem, we meet a large family of q-Hahn polynomials and we obtain in 
this way a combinatorial interpretation of this family of classical hypergeometric 
polynomials. Besides, we provide a Rodrigues type formula for these polynomials 
adapted to the present framework. For the second problem, we obtain the product 
formula by two different processes. One lies in the theory of classical hypergeometric 
polynomials and the other in the theory of finite symmetric spaces and spherical 
functions. The second approach also provides certain identities for the kernels in 
B2 which might be of independent interest. The results we present in this paper 
constitute the (/-analogue of those given in [Hj . The main idea is to identify certain 
Laplacian type operators on Grassmann graphs as hypergeometric type operators. 
This procedure will allow us to apply some results on classical hypergeometric 
polynomials which have appeared recently in [§]■ The paper [S] provides a new 
approach to the theory of classical hypergeometric polynomials which somehow 
lives between the theories developed by Askey and Wilson on one side and by 
Nikiforov, Suslov and Uvarov on the other. One of the main motivations for 
this paper is to show the efficiency of the point of view suggested in |Hj . 
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The organization of the paper is as follows. Let O be a finite-dimensional vector 
space over a finite field K and let G be the general linear group of endomorphisms 
of il. Let us consider the left regular representation p : G — * fi(Vx) associated 
with the natural action of G on the set X of linear subspaces of ft. Section ^ is 
devoted to describe a natural basis B of the intertwining algebra Endc(Vx). The 
Laplacian operators on Grassmann graphs are studied in Section [3 This is used 
in Section to identify these operators with some well-known hypergeometric type 
operators. Then we provide polynomic expressions and Rodrigues type formulas 
for the kernels of the operators in B. Finally, in Sections 01 and we give two 
different proofs of the product formula mentioned above. Section 21 uses the theory 
of classical hypergeometric polynomials while the proof given in Section lies in a 
characterization of spherical functions on finite symmetric spaces. 

After having written this paper, the authors were informed on the existence 
of Dunkl's paper which also identifies the kernels mentioned above as g-Hahn 
polynomials and studies similar relations for them. We note however that there also 
exist some significant differences between both papers. Indeed, our main product 
formula is not obtained in jS] while the Rodrigues formulas deduced from |S] (with 
non-ramified weights) are new. 

1. The object to study 

Let K denote the field ¥ q with q elements for some power of a prime q. In 
what follows, f2 will be a finite-dimensional vector space over K. If n stands for 
dimfi, we shall also consider the set X of linear subspaces of f2 and the sets X r of 
r-dimcnsional subspaces of £1 for < r < n. For any (x, y) E X x X, we define 

d(x,y) = dim (x/(x n y)) , 

d(x,y) = d(x,y) + d(y,x). 

Recall that, by Grassmann formula, we also have d(x,y) = dim((a; + y)jy). Taking 
d(x) — dim(x), we can write d(x,y) = d(x) + d(y) — 2d(x (~l y). The function 
d : X x X — > R + is clearly a graph distance on X and the same happens with 
the restriction d r : X r x X r — > W.+ of d on each X r . The distance d r imposes 
on X r an structure of distance-regular graph. These graphs are well-known in the 
literature as Grassmann graphs, see 0] for more on this. Besides, we shall also 
need to consider the vector space V of complex valued functions (p : X — * C and its 
subspaces V r made up of functions ip : X r — > C for < r < n. Notice that both V 
and V r are vector spaces over the complex field. If we consider the natural Hilbcrt 
space structure on these spaces, so that V = L2PC) and V r = L2(X r ), we clearly 
have the orthogonal decomposition 

n 

v = 0v r . 

1.1. Finite symmetric spaces. Before stating in detail the problem we want to 
study, we give a brief summary of results on finite symmetric spaces and spherical 
functions that will be used in the sequel. For further information on these topics 
see ^T] and the references cited there. Let G be a finite group acting on a finite 
set X. This action gives rise to the left regular representation p : G — > 6(^2 (X)), 
defined as follows 

(p(g)<p)(x) = fig^x). 
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Assume that the action G x X — > X is transitive, so that X becomes a homogeneous 
space. Then X is called a finite symmetric space with respect to the group G if 
the algebra Endc(£2(X)) of intertwining endomorphisms of L 2 (X) is abelian. 

Remark 1.1. EndG(^2(X)) is abelian if and only if p is multiplicity-free, see 

Now, if we are given two finite symmetric spaces Xi and X2 with respect to G, 
let us denote by p\ and P2 the corresponding associated unitary representations. 
Then we assign to each A in Hom(L2(Xi), L 2 (X 2 )) its kernel A : X2 x Xi — > C with 
respect to the natural bases. The mapping A — > A is clearly a linear isomorphism 
<J : Hom(L 2 (Xi),L 2 (X2)) -> L 2 (X 2 x Xi) with A and A related by 

Aip(x 2 ) = 2J X(x 2 ,x 1 )(p(x 1 ). 
X16X1 

Let HomG(i2(Xi), L2CX2)) be the space of intertwining maps for p\ and p 2 . If we 
compare A o pi(g) and p 2 (g) A written in terms of A, it is not difficult to check 
that A 6 HoniG(i2(Xi), L 2 (X 2 )) if and only if X(gx 2 ,gxi) = X(x 2 ,xi) holds for all 
.9 G G and all (xi,x 2 ) £ Xi x X2. That is, A is an intertwining operator for p\ and 
p 2 if and only if A is constant on the orbits of the action 

G x X 2 x Xi 3 {g, {x 2 ,xi)) 1 — ► (gx 2 ,gxi) G X 2 x X x . 

Now, assume we are given a transitive action G x X — > X of a finite group 
G on a finite set X endowed with a distance d. We say that X is a two-point 
homogeneous space when for any two pairs (xi,x 2 ), (y±, 2/2) G X x X satisfying 
d(xi,x 2 ) = d(yi,y 2 ), there exists g G G such that gx\ — y± and gx 2 = y 2 . 

Remark 1.2. An action G x X — » X is called symmetric when for any x\ : x 2 G X 
there exists g G G such that <?iei = 2:2 and g:E2 = x\. Any two-point homogeneous 
space X is clearly equipped with a symmetric action and in that case X becomes 
a finite symmetric space with respect to G. Namely, if the action of G on X 
is symmetric, then ^ (EndG(£2(X))) is a subalgebra of L2(X x X) made up of 
symmetric matrices, hence abelian. Finally, since the mapping ^ is an algebra 
isomorphism when Xi = X2, it turns out that X is a finite symmetric space. 

Let us write G for the dual object of G. That is, the set of irreducible unitary 
representations of G up to unitary equivalence. Let us consider the set 

G x = {tt £ G : Mult ff (p) ^ o}. 

Notice that if X is symmetric with respect to G, then every 7r G Gx satisfies 
MuhV (p) — 1 since p is multiplicity- free by Remark 11.11 This set allows us to 
decompose L 2 (K) into irreducible components 

L s(X) = L 2 (X) W . 

TreGx 

We denote by P v the orthogonal projection onto L^X)^. The kernel of P,r will be 
denoted by p„. The spherical functions on X are defined by 

^x, w = -jJW* G * (End G (MX))) , 
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where 7r € Gx and d(ir) denotes the degree of it. We shall also write Sx,tt for the 
associated operator in Endc(£2(X)) with kernel ipx,n- A slightly modified version 
of the following result can be found in Terra's book ^2 Th. 1 of Chapter 20] . 

Theorem 1.3. Let X be a finite symmetric space with respect to the finite group 
G and let ip G 'J' (EndG(L2(X))) ; then the following are equivalent: 

(a) There exists -k G Gx such that tp = ipx,n- 

(b) The function tp satisfies ip(xo,xa) = 1 for all xq G X and 

^{9x1, x 2 ) = ip{x 1 ,x )i/j{x ,x 2 ) = j^— 1 ip(x 1 ,gx2) 



IG 



seG X0 1 ""'gee,,, 

/or every x±,x 2 G X and where G Xo denotes the isotropy subgroup ofxo. 

1.2. Notation and results from q-combinatorics. We shall also need some 
results from q-combinatorics that we summarize here. Our notation will follow the 
book [S] by Gasper and Rahman. For some related results in q-combinatorics, the 
reader is referred to [7]. The q-shifted factorials are defined as follows 



(■«;<?)* 



fc-i 

3=0 



q 3 u) 



and 



fc-i 

]J(l-q- j u). 

3=0 



Then, the q-binomial coefficients can be written as 

(q;q) m (q m \q' 1 )k 



m 
m — k 



1 (q;q)k(q;q) m -k (q k ;q x )k 

Now we present some well-known combinatorial identities that will be used in this 
paper with no further comment. Let us recall some of the objects introduced above: 
K, fl, X r , V r , d, . . . Besides, let us consider the general linear group GL(n, K). That 
is, the group of endomorphisms of Q. Then, we have 



(1) 
(2) 



|GL(n,K)| = 
dim V r = 



(-l)"q(S)(q;q) n , 



Moreover, given Q<r\<r<r 2 <n and (xi, x 2 ) G X ri x X r2 , we have 



(3) 



,x G X r \ xi C x C x 2 





r 2 -ri~ 




. r — r\ . 



Eventually, we shall also use the q-multinomial coefficients 



q (q; q)n ■ ■ ■ (q; q)r 



with m = rj and 7*1, r 2 , 



{(an,..., a*) G JJX r . I ^ = 0^1 



1 Tk > 0. The combinatorial interpretation is 

k 



3 = 1 

In particular, given z G X r we have 



i=i 



(4) 



{u> G X„_ r J = z © u> j 



n, ■ 



= q r{r 



,T k lq 



T[q < 



l<3 



-r) 
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Let In(ri, r 2 ) be the set of all possible values of the parameter d(x 2 , x\) for x\ G X ri 
and X2 6 X,- 2 . Given t £ I n (ri, r2), the following identity follows from the relations 
above and will be very useful for our forthcoming computations 



(5) ^(x 1 ,x 2 ) eX ri xX r2 \ d{x 2l x 1 ) =t^ 



n 

t, r 2 - t, r\ — r% + t, n — r\ — t 



d 



Remark 1.4. We shall also use the notation (m, u 2l ■ ■ ■ , Ud', q)k = \\_{ u j'i 



=i 



1.3. The basis of the algebra Endc(V). Let G be the general linear group 
GL(n, K) of endomorphisms of fl considered above. This group acts naturally on 
the set X of linear subspaces of the vector space fl. The orbits of this action are 
the subsets Xo, Xi, . . . , X„ of linear subspaces of dimensions 0,1, ... ,n. 

Remark 1.5. The action of G on X r preserves d r and X r is clearly a two-point 
homogeneous space for each < r < n. In particular, by R emark 1 1.21 X r becomes 
a finite symmetric space with respect to G. Besides, Remark ll . II gives that the left 
regular representations of G into V r are multiplicity-free for each < r < n. 

Given any two integers < n, r 2 < n, we shall identify each space Hom(V ri , V r2 ) 
with a subspace of End(V) by right multiplication by the orthogonal projection from 
V onto V ri . Applying the same identification for the intertwining operators, we 
obtain the following decompositions 



End(V) = 0Hom(V ri ,V r2 ), 

T\ ,T2 

End G (V) = 0Hom G (V n ,V r2 ). 



The algebra End(V) is a Hilbert space with respect to the Hilbert-Schmidt inner 
product and the direct sums given above become orthogonal decompositions with 
respect to this structure. Besides, we know that the kernel of any intertwining 
operator A £ HomG(V ri , V r2 ) is constant on the orbits of (g, (x 2 , i — * (gx 2 ,gx\). 
These orbits are completely determined by the parameter d(x 2 ,xi). In particular, 
the kernel of A can be regarded as a function A : I rl (r 1 ,r 2 ) — > C, where the index 
set l n (ri,r 2 ) was considered above 

In(7i,r 2 ) = ^ ] d{x 2 ,x 1 ) | x 1 6 X ri ,x 2 E X r2 j = jo V (r 2 - n) < t < r 2 A (n-n)j. 
Here A stands for min and V for max. Therefore we have 
Aip(x 2 ) = ^ Hd(x 2 ,x 1 ))(p(x 1 ) 

for any ip G V ri and 

dim (Hom G (V ri ,V r2 )) = |l„(n,r 2 )| =1 + N(ri,r 2 ), 

where N(ri, r 2 ) = r± A r 2 A (n — r\) A (n — r 2 ). Reciprocally, any A : I n (ri, r 2 ) — » C 
determines an operator A G Home ( V ri , V r2 ) . Moreover, since V r is multiplicity 
free, Schur lemma gives that the dimension of Homo(V ri , V r2 ) is the number of 
irreducible components that V ri and V r2 have in common. In particular, we have 
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• V r has 1 + N(r, r) = 1 + r A (n — r) irreducible components. 

• The number of irreducible components that V ri and V r2 have in common 
is the minimum of the numbers of irreducible components of V ri and V r2 . 

Therefore, there exist a family of inequivalent irreducible unitary representations 
ir s : G — > B{Tis) such that, if we denote by V r ,s the G-invariant subspace of V r 
equivalent to H s , the left regular representation p r : G — > B(V r ) and the Hilbcrt 
space V r decompose into irreducibles as follows 

rA(n-r) rA(n-r) 

Pr— ® and V r = V ns . 

Moreover, the representations of G into V ri)S1 and V r2iS2 are equivalent if and only 
if si = s 2 . Finally we note that 

(6) dim V ryS = dimV S)S = dimV s — dimV s _i = 

Here we assume by convention X_i = 0, so that dimV_i = 0. The last identity 
in JSJ follows from relation By Schur lemma we know that Hom G (V ri;S , V r2jS ) 
is one-dimensional. In summary, we have obtained an orthogonal decomposition of 
the algebra of intertwining operators End G (V) into one-dimensional subspaces 

N(n,r 2 ) 

End G (V)= Hom G (V ri , s ,V r2 , s ). 

0<ri,r2<n s— 

This decomposition provides a natural basis of the algebra End G (V) which will be 
the object of our study. Namely, taking a non-zero element A!>' r2 in each space 
HoniG(V ri , s ,V r2!S ), we obtain a basis B of the algebra End G (V). Our definition of 
A^ 1 ' r ' 2 is still ambiguous since we have only defined it up to a constant factor. We 
shall precise this below. As it was announced in the Introduction, the aim of this 
paper is to provide several expressions for the kernels A^ 1 ,T2 and to give an explicit 
formula for the mapping product A!?' 1 " 3 o A^ 1,r2 . 





n 


n 


s-1. 



2. LAPLACIAN OPERATORS ON GRAPHS 

In this section we deal with some Laplacian type operators which will be useful 
to identify certain difference equation satisfied by the kernels of A^ 1 ' 1 " 2 in the usual 
rank of parameters for r\,r% and s. We begin by recalling some general results for 
Laplacian operators on graphs. Then we focus on the Grassmann graphs. 

2.1. General results. Let X be a finite distance-regular graph and let d be the 

distance on X. Assume there exists a finite group G acting on X such that the graph 
X becomes a two-point homogeneous space with respect to G. This structure on X 
allows us to define two Laplacian type operators on the vector space V of complex 
valued functions <p : X — > C. First, given i£X, we consider the set 

S 1 (x) = {yeX| d(x,y) = iy 

The regularity for the distance imposed on the graph X implies that the cardinality 
of the set Si(x) does not depend on x. This cardinality val(X) is called in the 
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literature the valence of X. The graph Laplacian Cx ■ V — > V is the operator 
defined as follows 

£ x tp(x)= Y i^iy) - P( x )) = Y <p(y)-val{X)<p(x). 
yeSi(x) i/€Si(x) 

Second, let us consider a subset T of G satisfying the following properties 

• T = T" 1 . 

• T = gTg- 1 for all g G G. 

• d(x, hx) < 1 for all x G X and all ft G T. 

• There exists x G X such that T G x , the isotropy subgroup of x. 
The group Laplacian Ct.x '■ V — > V is defined as follows 

£ T ,x^<» = Y (v( hx ) ~ ^( x )) = Y ^ hx "> ~ \ T \f( x )- 

heT hET 

Both the graph and the group Laplacians are self-adjoint operators with respect 
to the natural inner product on V. This is an easy exercise that we leave to the 
reader. Besides, recalling that an endomorphism of V is an intertwining operator 
if and only if its kernel is constant on the orbits of the action of G on X x X, it is 
not difficult to check that both the graph and the group Laplacians belong to the 
intertwining algebra Endc(V). Now let T be a subset of G satisfying the properties 
above. Then, given x G X and y G Si(x), we define 

= j/i G T | hx = xj, 

T x , v = \h<ET\hx = y y 

From the properties of T it follows that gTxg" 1 = T gx and gTx^g -1 — T gx ,gy 
for all g G G. In particular, since the distance- regular graph X is assumed to be a 
two-point homogeneous space, the numbers 

7o (T,X) = |T X |, 
7i(T,X) = \T X , V \, 

do not depend on the election of x G X and y G Si (a;). We now state some basic 
results on these Laplacian operators that will be used in the sequel. Notice that 
the fourth condition imposed on T implies that 71 (T, X) > 0. 

Lemma 2.1. |T| = 7o (T, X) + val(X) 7l (T, X) and £ T ,x = 71 (T, X) £ x . 

Proof. Obviously we have 

T = T x U [ |J T x . y 

yeSi(x) 

with disjoint unions. Hence, the first assertion follows. Besides, we notice that 
£t,x^(x) = Y {<P( hx ) - V( x )) = Y Y iv( hx ) ~ 

heT yeSi(x) h£T Xiy 

By definition, the last expression is 71 (T, X) £x<fi(x). This concludes the proof. □ 
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Lemma 2.2. Let W be a G-invariant subspace of V. Then, the group Laplacian 
£t,x preserves W. Besides, if W is irreducible, there exists a complex number p 
depending only on the representation of G into W such that 

£t,X| w +^1w = 0. 

Proof. Let p : G — > £>(V) be the left regular representation associated to the action 
of G into X. Then, for any ip 6 W, we can write 

heT 

The first claim follows from the relation above. The second is a consequence of Schur 
lemma. Namely, if W is irreducible and it : G — > £>(W) denotes the restriction of p 
to W, we have 

£t,X| W = - l w )- 

heT 

Therefore, since this is an intertwining operator with respect to tt, Schur lemma 
gives that £t,X| W + M lw = with p depending only on the representation n. □ 

2.2. Laplacian operators on the graphs X,.. In this paragraph we return to the 
study of the intertwining algebra Endc(V) described in Section ^ hi particular, 
we shall work with the general linear group G of endomorphisms of f2 and the 
Grassmann graphs X r for < r < n. We denote by C r : V r — > V r the graph 
Laplacian on X r while £x,r '■ V r — * V r stands for the group Laplacian. If rk(A) 
denotes the rank of a mapping A, we consider the subset T of G defined as follows 

T = [g e G | rk(g - 1) = 1, (g - l) 2 = o} 

= |l + w®a| aeff\{0},weKer(a)\{0}}, 

where u ® a £ End(fi) is given by (cj ® a)(w ) = a(cj )w. We need to check that 
the subset T satisfies the properties introduced before the definition of the group 
Laplacian. But this is an easy exercise that we leave to the reader. The main 
results of this section are summarized in the following theorem. We shall also need 
to consider the operator C ri ,r 2 '■ HoniG(V ri , V r2 ) — > Home ( V ri , V r2 ) defined by 
the following relation 

Theorem 2.3. The Laplacian operators considered above satisfy: 

(a) We have 

ll (T,X r ) = q n - 2 (q-l) and C T , r = q' l - 2 (q - l)C r . 

(b) The mappings A G Hoitiq (V n , V r2 ) satisfy 

C r2 o A = C ri , r2 A = A o C ri . 

(c) Given ip € V r . s and A G Home (V ri , s , V r2;S ), there exists p s (n) such that 

C r ip(x) + p s (n)(p(x) = 0, 
C ri . r2 A + p s (n)A = 0. 



10 



MARCO AND PAR.CET 



(d) If A £ Home (V ri , V r2 ) has kernel A : In ; ?"2) — > C, then 

has kernel A' : I n (fi, r%) — ► C given by 

*'(*) = b ri , r2 (i)(A(t + 1) - A(i)) + c ri , r2 (i)(A(f - 1) - A(i)) 
where, given {x\,X2) 6 X ri x X r2 tuii/i 9(x2,a;i) = t, we /iai>e 

b n,r 2 (*) = {y G X r2 | x 2 ) = 1, d(y, x±) = t + 1 J 

c r u r 2 (t) = |y G X^, | d(y,x 2 ) = l,d(y,xi) =t- lj 

(c) T/ie following expressions hold 



b T1 ,r 2 (^) 



,(*) 



( 9 -l) 2 



(g -l) 2 

The proof of Theorem 12 . 31 rea uires several auxiliary results. We shall state and 
prove these results as they are needed. Given x E X r , we consider the sets 

Si(s) = {y GX r | d r (x,y) = l}, 

Si(x) = G X r _i X fi| z C x,lo x\. 

Lemma 2.4. Given any x E X r , £/ie mapping 

(z,lj) e Si (x) i — >i/ = zeiwe Si (a) 

«s surjective, z — x Dy and there exist q r ~ 1 (q — 1) possible lo for each y E Si(x). 

Proof. Given x,y E X r we have d r (x,y) = r — d(x R y). Hence d r (x,y) = 1 is 
equivalent to d(x PI y) = r — 1. In other words, y = z © Ko; with 

z = x Dy and w £ y \ z. 

Hence we have lo x, (z,lo) E Si(x) and there exists 

\y\z\ = q r -q r - 1 

possible elections for the vector lo. Therefore, the proof is completed. □ 

Lemma 12.41 allows us to compute the valence of the Grassmann graphs X r . 
Namely, since |X r | = dimV r , we can apply (J2J to obtain for x E X r 

7 val(X r ) = Si (a;) = -— j-7 — = , — —, -r = -, ^ . 

Lemma 2.5. Given j = l + w®a£T, we /iaue gx = x iff x C Ker(a) oc w e s. 

Proof. If X C Ker(a), it is clear that gx = x. Besides, if x is not a subspace of 
Ker(a) but lo £ x, we have x = x' © Ku/ with x' = x R Ker(a) and u/ E x \ Ker(a). 
This gives 

gx = x' © Kp(c/) = x' © K(a/ + a(w')w) = x ' © = x - 

Reciprocally, if gx = x and x does not belong to Ker(ct), we take lo" G x \ Ker(a) 
so that = cj" + ce(cj")cj G x. Therefore, we conclude that lo £ x. □ 
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"n — r~ 




.1. 


9 


. 1 _ 





Now we combine the expression for the valence of X r given in JJJ with J2J and 
Lemma 12.51 to obtain the value of 7i(T,X r ) 

val(X I .) 7l (T,X r ) = |{ 5 gT| 5 x^.t} 

= |l + w ® a| a efl* \ {0},lu G Ker(a) \ x,x £ Ker(a)} 

){q n ~ 1 -q r - 1 ). 

Dividing on the right hand side by the value for val(X r ) given in (J7J, we obtain the 
identity 7i(T,X r ) = q n ~ 2 (q — 1). This proves the first assertion of (a) in Theorem 
12.31 The second assertion follows from Lemma \'2. II On the other hand, since any 
A G HoixiG(V ri , V r2 ) commutes with the action of G, we have 

£-T.r 2 O A = A O £T,ri • 

Therefore, (b) in Theorem 12 . 31 follows from (a). Moreover, (c) is a consequence of 
Lemma 12.21 and (a). To prove (d), we take (x\,x 2 ) G X ri x X r2 with d(x2,xi) = t. 
Then we have 

A'<p(x 2 ) = i A( p(y) ~ A V(X2)) 

yes 1 (x 2 ) 

= E E (Md(y,x 1 ))^X(d(x 2 ,x 1 )))c P (x 1 ). 

y£S 1 (x 2 ) x 1 eX ri 

In particular, we can write 

X'(d(x 2 ,x 1 ))= Yl (A(%n))-A(fl(u,n))). 

Then (d) follows immediately from this. Finally, it remains to see (e). The proof 
requires two combinatorial lemmas. Let us notice that, given {x\,x 2 ) G X ri x X r2 
and y G Si(x2), we have 

(8) d(x 2 ,xi) - d(y,xi) = d((xi + x 2 )/xi) - d((xi + y)/x\) 

= d((x! + x 2 )/{ Xl + z)) - B(( Xl + y)/(x 1 + z)), 

with z = x 2 H y. Besides, recalling that y G Si (x 2 ), it is not difficult to check that 
both dimensions appearing on the right hand side of JHJ are either or 1. This 
remark will be used in the following results. 

Lemma 2.6. Given (x\, x 2 ) G X ri x X r2 , (z, u) G Si (2:2) and y — zffilw G Si (x 2 ), 
the following assertions are equivalent: 

(a) d{y,xi) = d(x 2 ,xi) + 1. 

(b) x\ + z = x\ + x 2 and uj X\ + x 2 . 

(c) xi (~l x 2 ^ z and uj ^ x\ + x 2 . 

Proof. Following JHJ and the remark after it, we deduce (a) is equivalent to (b). 
On the other hand, since 



d((xi + x 2 )/(xi + z)) = 1 - d{x\ n x 2 ) + d(xi n z), 
it follows that (b) is equivalent to (c). Therefore, the proof is completed. 



□ 
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Notice that z is a [r% — l)-dimensional subspace of X2- Besides, given X2 G X r2 , 
the quotient mapping 7r : X2 — ■> ^2/(^1 H x%) provides the following identity 

C X2 I d(z) = ti— \,x\r\x<x C = c X2f(x\C\X2) I d(z') = 8(x2,Xi)— lj 

In other words, the number of subspaces 2 of £2 satisfying (c) is 



T2 




C?(x2,Xl) 




r 2 ~ 




d(x 2 ,x 1 )' 


r 2 -l- 


9 


.9(a;2,a;i) - 1. 


9 


. 1 _ 


9 


1 



Combining this with Lemmas 12 .41 and 12 . 61 we easily get the following expression for 



which simplifies the one given in Theorem 12.31 



.(*) = 



r 2 -l 



(9-1) 



,ri+t 



Lemma 2.7. Given (xi,x 2 ) G X ri xX r! , (z,w) G Si(x2) and t/ = z®Kw G Si(x2), 
the following assertions are equivalent: 

(a) d(y,xi) = d(x 2 ,xi) - 1. 

(b) Xi + z 7^ xi + X2 and u> E xi + z. 

(c) xi (~l X2 C z and lu G x\ + z. 

Moreover, when them hold we have d(x\ -\-z) = 9(xi + X2) — 1 and z = (xi + z) l~IX2. 

Proof. By JHJ and the remark after it we deduce the equivalence between (a) and 
(b). The equivalence between (b) and (c) follows again from the identity 

d((xi + x 2 )/ (xi + z)) = 1 - d{x\ n x 2 ) + d(xi n z). 

Relation d(x\ + z) = 9(xi + X2) — 1 is immediate from (b). The last claim follows 
from the modular law (xi + z) n X2 = (xi + z) n (X2 + z) = (xi n X2) + z = z. □ 

Finally, arguing as we did after Lemma 12.61 we easily obtain from Lemmas 12.41 
and 12. 71 the following identity 



,(*) 



-n+t-l 



^2 — 1 



t>-2-1 



(9-1) 



This is the identity given in (e) and the proof of Theorem 12.31 is completed. 

3. The kernels in Hom G (V riiS , V r2;S ) 

In this section we obtain explicit formulas for the kernels of the operators Ag 1 ' 1 " 2 
in terms of the basic hypergeometric function. More concretely, it turns out that 
these kernels (when regarded as functions of q with t G In(^ij^2)) are given by 
the so-called g-Hahn polynomials. After that, we shall also provide Rodrigues type 
formulas for these kernels adapting the techniques developed in [H] . The basic idea 
consist in showing that the operator C ri >r2 introduced above can be identified with 
the hypergeometric operator studied in j^j. 

3.1. Preliminaries. As we have pointed out, we shall need some results from the 
theory of basic hypergeometric polynomials appearing in 9 . However, since the 
paper 9 considers a great variety of hypergeometric type operators, we summarize 
and re-state here those results from |Hj that will be useful in the sequel. To be 
more precise, we shall formulate the main results from ^] in the particular case of 
the (so-called there) geometric canonical form. Let us denote by ¥ r [x] the space of 
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complex polynomials of degree < r in one variable. Given a polynomial / £ P r [x] 
and any q ^ 1, we define the linear operators 

f( q ^u) + f(q-^u) f( q ^u)-f(q-^u) 
S/(«) = g and D/(u) - (g i /2 _ g -i /2)u • 

It is not difficult to check that S : ¥ r [x] -> P r [x] and D : P r [ar] —> P r -i[x]. Moreover, 
these operators can be extended so that S,D : M(C*) — > A4(C*), where .M(C*) 
stands for the space of meromorphic functions on C\ {0}, see Section 2.3]. Now, 
given a £ P 2 [x] and r € Pi [x] by 

cr(x) = a 2 x 2 + aix + ao and t(x) = fi\x + 

we consider the hypergeometric operator 

L = o-D 2 + tSD. 

The following result has been adapted from [H] according to our aims. 

Lemma 3.1. The hypergeometric operator L : P r [x] — > P r [x] satisfy: 
(a) Le£ ms consider two polynomials 

X + (u) = 7^^+7^ + 70, 
= 72~m 2 +7i"u + 7o, 
with the same value at u = and let us parameterize a and r as follows 

X + (m) = cr(u) + ^ ur(u), (t(u) = , 

_ n , , g- 1/2 -g 1/2 , v . . X + {u)-X-(u) 

X (u) = *(«) ^ ut(u), r(«) = (g _ 1/2 _ gl/2)u - 

Assume that the following numbers are pairwise distinct for < k < r 



~ h. j 2 ,y / 2 



a'*'" - o 



7 + g (i-fe)/2_ 7 - 9 (fe-l)/2 



(gl/2 _ g-l/2)2 

TTien i/iere exisi eigenf unctions fk € P& [x] wi£/i degree k satisfying 

L /fe( w ) + = 0. 

(b) Lei /fe fee as above and let dkfk be the main coefficient of fk-' 

• Given uo such that 7^ Uq + 7^ uo + 70 = 0, we /iaue 

AW-*Ag(g^^^)n(.-^). 

• Given uq such that r J2 u a + 7r u o + 7o = 0, we Ziaue 

/fe(u) = dkfk V ( TT _ 1/2 , -To 7 7 — ) TT(u - <T*uo)- 

(c) Moreover, assuming above that /fc(uo) ^ 0, we obtain: 

• If + 7^1*0 + 7o = and fk(uo) 7^ 0, we /iaue 

/*(«) = Muo) J2 { II i z g - ( i-H) -p^j J IK- - 9 t*). 
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tfl-z u o + 7i u o + 7o = and fk(uo) ^ 0, we have 



j=0 i=0 y i=0 

(d) If X ± ( u ) = (1 — Co = ' lt )(l — £i u )> we obtain the q-Hahn polynomials 



• /*(«) - / fe (i/e + ) 3^2 

• /fc(u) = 3^2 

• /*(«) = /fe(l/^) 3^2 

• /*(«) = /fc(l/4T) 3^2 



sVAo : S7£r 

g fe , g 1 -*^^/^^" 
g fc , l/^-u, g^tf^/M 

e + /er, tf/sr 



g- 1 , T 1 



g- 1 , T 1 



g \ g x Ci u 



g 1 , g u 



Remark 3.2. The role of q in this paper is played by g _1 in UJ. 

Remark 3.3. The notation for the basic hypergeometric function follows (HJ- 

We shall also give a Rodrigues type formula for the kernels in Hom.G(V riiS , V r2jS ). 
To that aim, we state below the Rodrigues formula given in [2] which corresponds 
to our problem. That is, the one for the geometric canonical form. 

Lemma 3.4. Let p G A4(C*) be a function satisfying the functional equation 

p{u)x + {u) = p(q- 1 u)x~(q~ 1 u). 

Let pj G M.(C*), with j > and po = p, determined by any of the recurrences 

p ]+1 (u) = Pj (q^u) X + (q-( j - 1)/2 u), 
p J+1 (u) = p 3 {q-^u)x-{q {j - 1] l 2 u). 
Then, the following Rodrigues formula holds for the eigenf unctions /o, /i, . . . , f r 



k-l 



Ylil 1 ] - ^k))p{u)fk{u) = dkfki Y[ — I 



fe " 1 a (k-j)/2 _ Jj-k)/2 . 

^ k p k (u). 



3=0 



3=0 



1/2 _ q -i/2 



3.2. Polynomic expressions. In this paragraph we express the kernels of the 
operators of B in terms of the basic hypergeometric function. Let us recall that, 
given < ri, r2 < n, the parameter N(t*i, r^) takes the value 

r\ A r2 A (n — ri) A (n — r-z) 

and the dimension of HomG(V ri , V r2 ) is 1 + N(ri,r2). In particular, we can define 
the linear isomorphism $ : F N ( ri I , 2 ) — > HoniQ ( V ri , V r2 ) given by 

A^(x 2 )= J2 f(q- d(X2 ' xl} )^i), 

iiex n 

for A = $/. In other words, the kernel A : l n (ri, r^) — » C of $/ has the form 

Kt) = /(«-*)• 



Then we define L 



^(ri,r 2 ) by the relation L riiT . 2 = $ o£ riiT . 2 o$. 
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Lemma 3.5. Let us consider the following polynomials in ¥2[x] 
X +(u) = q^-^a-q^u^l-q^u), 
X~(tt) = q ri - r2 - 1/2 {l-u){l-q r2 - ri u), 

and let us parameterize a E P2 [%] and t E Pi [x] as follows 



q -i/2 _ q i/2 

X + {u) = o-(u)-\ ut(u), a(u) 

g-l/2 _ gl/2 

X (u) = a(u) ut(u), t(u) 



X + (u) +X~{u) 
2 _ ' 
X + (u)-X ( u ) 



2 ~' v ~ 7 ' '"~ J ( g -Va-gi/2)u" 
Then, if S and D stand for the operators defined above, we have 

L ri;J . 2 = crD 2 + rSD. 
Proof. Given / E P N r ri)J . 3 ), we have 
trDVCg^J + rSD/Cg-*) 



g-t-1/2 _ g-t+1/2 



q 1 1 — q 1 



q -t _ g-t+l 



r(?-*) [/(g-*- 1 ) -/(?"*) , /(?"*)- /(<T t+1 ) 



g-t _ g-t + l 



2tr( g - t ) + (g- 1 / 2 - q^^q-tTjq-t) 

2{q- l)2g-2t-3/2 

Mg-j) - (a' 1 / 2 -g 1 / 2 )g- t r(g- t ) 

2(9- l)2g-2t-l/2 
24+3/2^+^-4) 



(A(t + 1)-A(f)) 
(A(t - 1) - A(t)) 
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2t+l/2 -/ -t\ 

(A(t + 1) - A(t)) + q X ^ ' (Xit 1) - A(t)). 



(g-1) 2 ^ 7 w ' (g-1) 2 
Now, applying Theorem 12.31 the last expression equals 

b rur2 (t)(X{t + 1) - A(t)) + c ri , r2 (t)(A(f - 1) - A(t)). 
Applying again Thcorcm l2.3l the proof is concluded by the definition of h rijT2 . □ 
Theorem 3.6. The spaces V r s are C r -eigenspaces with eigenvalue —fi s (n), where 

gTl— S+l _ 1 

Ms (n) = (g s - 1) (g _ 1)2 ■ 

Moreover, the operators A£ 1,r2 E B satisfy 

£ ri , r2 A^^+M s (n)A^^ = 0. 
On £/ie other hand, if we consider the polynomials /J 1 ' 1 * 2 E P s [x] determined by 

f r s 1 ' r2 (q- t ) = K ur2 (t), 
then /J 1 ' 1 " 2 «s a q-Hahn polynomial of degree s given by 



(h,(i)) /; i ' r2 w = /; i ' r2 (i) 3^2 



g s , u- 1 , g™"^ 1 



(h,(2)) /r i,ra («)=/r' r, (9" ra ) 302 
(h,(3)) /r ir2 w = /r i,r2 (« ri - n ) 302 



gn-r 1; g r 2 



g s , g r2 u, g«- s +! 



g _1 , g r3 - ri - 1 u 



g ri , g r2 



q s , q n - ri u, g«- s +! 

q n-n } q n-r 2 



q-\ q- 1 
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q s , q^-^u' 1 , q n -" +1 



q 1 , q 1 u 



(H,(4)) fl 1 ' r2 (u) = f: i ' r2 (q ri - r2 ) 3</>2 ( .. „ 

Proof. Notice that (i s (n) are pairwise distinct for < s < N(ri,r2), since 

n+l-si-s 2 _ 1 

H.i (n) - M S2 (n) = g S2 (q Sl " S2 - 1) ^ (g _ 1)2 • 

Therefore, we know from Lemmas 13.11 and 13 . 51 that 

n-s+l _ i 

(l-O (g _!)2 I 0< S < N (^1^2) 

is the family of eigenvalues of L rijI , 2 . In particular, it turns out that this family 
is the family of eigenvalues of C r when 0<s<rA(n — r) and of C ri ^ r2 when 
< s < N(r!,r 2 ). By Theorem 12.31 we deduce that, for any < r < [n/2] — 1, 
all the eigenvalues of C r are eigenvalues of C r+ i and the operator C r+ i has one 
more eigenvalue associated to the eigenspace V r+ i, r+ i. Applying a simple induction 
argument, we know that the eigenvalue —fi s {n) of C r is associated to the eigenspace 
\ r ,s- As a particular case, we obtain the relation 

£ P1 , ra A?' ra +M.(n)A?' r * = 0. 

Once we have identified the eigenvalue corresponding to the operator A^ 1,r2 , the 
given expressions for the polynomial /J 1 ' 1 * 2 in terms of the g-Hahn polynomials 
follow easily from Lemma f3. II This completes the proof. □ 

Remark 3.7. Let us denote by <9 S /I 1 ' r2 the main coefficient of /J 1 ' 1 * 2 . Then, by 
looking at the main coefficients of the expressions given in Theorem 13.61 it is not 
difficult to check that the following relations hold 

V(n-r 2 )-(l) (g"- ri ,g r2 ;g~ 1 )« 
(^.^ig' 1 ). 

_ ri i (<7"- ri ,s- n - r2 ;<r 1 ) s 





(1) 


= d.fr 








= d,f? 


r2 q 






= B a f? 


r 2q 


fl ur2 (q ri - 




= d.f? 


T2 ^ 



Remark 3.8. The basic hypergeometric series in Theorem 13.61 must be truncated 
at degree s. That is, the terms of degree > s must be ignored. This is a consequence 
of the term q s which appears in any of them. For instance, we have 



q s , u _1 , q n - s+1 
q n - r , q s 



q-\ q^-'u 



/•'•-rii \ • ( M >Q n >Q )fc fc(s-7— i) 7 ,fc 



fc=0 

which is a truncated 2<pi series. Now, evaluating at u = q~ l for some t G s) 
and applying the g-Gauss summation formula, we easily obtain 

(„s— i — l. „— l\ 
Js W )-Js W ( q n-r. q -l) t ■ 
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Similarly, we have for t£l„(n-s,r) 

(„r+s—n—X. fl—l} 

rn — s.r f — 1 \ rn — s.r/-i\ W 1 il 

Is W )-Js W {q r. q -l )t ■ 

At this point, the identity 

/P' r2 N = s(r 2 -r 1 ) (Q ri ,Q n - r2 ;Q- 1 )s fP^jq^u) 
fP' r2 (l) q (q^ jq r 2;q -l )s f p^ {1) 

follows from certain transformation formulas for the basic hypergeometric series 
3</>2, see p. In Section we shall provide an alternative (combinatorial) proof of 
this identity. Therefore, we prefer to omit the details of the proof just sketched. 

3.3. Rodrigues formula. In this paragraph we provide a Rodrigues type formula 
for the kernels X^ 1 ^ 2 of the operators in B. More concretely, given any two integers 

< ri,r 2 < n and < s < N(ri,r 2 ), we shall study the eigenfunctions /J 1 ' 7 " 2 
defined above by the relation 

f?' r *(q- t ) = \™(t). 

As it was noticed in |5] , the Rodrigues formula provided by Lemma 13.41 is not 
unique since the given functional equation has multiple solutions. Hence, the main 
difficulty will be to choose the right solution of the functional equation according 
to our further purposes. Following Lemma 13.41 let us consider p ri > r2 £ A4(C*) 
satisfying the functional equation 

(9) p r ^(u) x + (u) = ^'^(r^lrrM, 

with \ + and \~ determined by Lemma 13.51 Moreover, let 

{p?' r2 \ 0< S <N(n,r 2 )} 
be the family of functions in A4(C*) defined by any of the recurrences 

P :x?(u) = ^ r2 (<? i/2 ")x + (T (s - i)/2 «), 

1 ' P?+?(u) = P 7^{q- 1/2 u) X -{q {s - 1),2 u), 

where Pq 1,T2 — p ri,T2 . Then, implementing in Lcmma l3.4l thc eigenvalues provided 
by Theorem 13.61 it is not difficult to see that we obtain the following Rodrigues 
formula for the eigenfunctions /J 1 ' 7 " 2 

(q i q )s 

Remark 3.9. Following Section 4.5 of [Hj, the easiest solution to the functional 
equation Q and the recurrences I|1U|I is given by 



g ri ' r2 (u) 



(q r ^u,q n - r ^u;q- 1 ) c 



( n -l + s/2„. n r 2 -r 1 -l + s/2„.. n -l\ 
gl-i,r2t u \ _ q s( ri -r 2 -l/2)(q "■'.! " ■ '/ > 



[ q r2-s/2 u ^n-r 1 -s/2 u]q -l^ oo 

Although the system of functions given in Remark 13.91 provides the simplest 
Rodrigues formula for the eigenfunctions /J 1 ■ T2 , it is not the most appropriate for 
our aims. Namely, let us analyze the singularities of the function g Vl ' V2 . If we take 
uq = q~ l for some f £ Z, then the function g ri,r2 has: 
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• A double pole at uq when 

V (r 2 - n) < t < r 2 A (n - n). 

• A simple pole at uq when 

A (r 2 -n) <t <0V(r 2 -n), 

^2 A(u-ri) < t < f2 V (n — ri). 

• A non-vanishing regular point at uq when 

t < OA (r 2 -n), 
i > r 2 V (n — ri). 

The function g ri ' r2 does not have any other zeros or poles in C\ {0}. In particular, 
it turns out that the function g Tl ' T2 has singular points at q~ l with t belonging to 
the domain 

I« (71,7-2) = {d(x 2 ,xi) I xx e X ri ,x 2 e X r2 | = |o V (r 2 -n) <i < r 2 A (n-ri)|. 

Obviously, if we want to apply Rodrigues formula |R s (ri,r 2 ) I, we need regular 
solutions of the functional equation © at q~ f for t G I n (ri,r 2 ). Any other system 
of solutions can be constructed by taking 

p ri - r2 (u) = k(u)g ri - r2 (u), 

with k being meromorphic in C \ {0} and satisfying k(u) = k{qu). Then, the 
functions p^ 1 ' 7 " 2 arise from p ri ' T2 by the recurrences (|10|) . The choice of such a 
function k is equivalent to the choice of an elliptic function E(z) = fc(e 27rlz ) with 
periods 1 and 

2m 

sec 9 for further details. Hence, we need to find a function k £ Ai(C*) satisfying 
k(u) = k(qu) and having double zeros at I n (7i,?*2)- To that aim, we consider two 
complex numbers £ and r\ satisfying the conditions 

i) The product (j\ equals q n ~ r ^ +r ^ . 

ii) Both £ and rj are not of the form q for some integer t. 
Then, the function 

nn , ,x_ (g^, g"^; q- 1 )^ (q- r '- 1 u- 1 ,q- n + r i- 1 u- 1 ;q- 1 ) 00 

(11) K rir2 (U)—a rir2 , . _ _ _ _ 

{£u,r]u;q ^ (g ^ A u \g ^ A u x ;g ^00 

satisfies the required properties. Indeed, the condition fc(u) = k(qu) can be easily 
checked with the aid of property i). On the other hand, it follows from property ii) 
that k has no poles in C \ {0}. Therefore, q~ l is a double zero of k for any integer 
t and k has no other zeros. In particular, the function 

n ri,r 2 _ u n ri ' r2 

is regular in C \ {0}, non- vanishing in l n (ri, r 2 ) and only vanishes in q~( z \M I, i,i- 2 ))^ 
Remark 3.10. Notice that p Vl ' T2 is determined up to a constant a ri:T2 . 
Theorem 3.11. Given < r\,r 2 < n, there exists a family of functions 

{p?' r2 I 0<«<N(ri,ra)} 
in M(C*) satisfying the following properties: 
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(a) The function p ri ' r ' 2 — Pq 1 ' 1 ' 2 solves the functional equation lfp)l. 

(b) The function p Tl ' r2 is regular in C \ {0} and vanishes in 

(? -(Z\I„(r 1 ,r 2 ))_ 

(c) Given any integer t £ In(ri,r 2 ), w have 

n r i' r 2( n -t\ _ t{ ri -r 2 +t+l) [ n 

H yq ' q [t,r 2 -t,n-r 2 + t,n-ri-t 

(d) Each function p^ 1 ' T2 arise from p Tl ' r2 and the recurrences . 

(e) Each function p r s 1,r2 is regular in C \ {0} and vanishes in 

„-(Z\In-2a(r-i-s,r 2 -s))-s/2 

(f) Given any integer t 6 l n —2s(fi — s,r 2 — s), we have 



! (q 



-4-s/2\ _ 



n — 2s 

.t,r 2 — s — t, 7*1 — r 2 + i, n — n — s — 
with ip'^ 1 ' 7 ' 2 given by 

ip r /> r2 (t) = g ^i-r 2 -i/2)+4(ri-r 2 +t+i)^n. g -i^ 

Proof. Our choice for p Tl ' V2 will be k ri! r 2 Q ri ' r2 with k ritT2 given by and the 
constant a ri ^ r2 to be fixed. Properties (a) and (b) have already been justified. To 
prove (c), we observe that the functional equation (PJ can be rewritten as 



(12) r v, , r v , x+(q . t) 

when t and t + 1 belong to I n (Vi,r2)- Therefore, if we see that the function 



r 



! (<r 4 ) 



,4(7-1 -r 2 +t+l) 



„ t, ra — i, ri — r-2 + i, n — n — t J 9 

satisfies we will have p ri ' r2 (q^ t ) = Pr 1 ,r- 2 l ri ' r2 {q^ t ) f° r an Y * € I n ('"i)'*2) and 
some constant (3 ri ,r 2 ■ Then, property (c) follows by taking the appropriate constant 
a rit r 2 . Let us show that -f Tl ' r2 satisfies the functional equation l|12|) 



7 



>-{q 



(<?; q)t(q; q)r 2 -t(q; q) ri -r 2 +t(q; q) n - ri -t 
r L "*{q 



' < r--t' i \ n -{ r i- r 2+2t+ 2 ) ( 1 q )( 1 q 



ri -r 2 +t+l^ 



g7- 2 — ri— 4— 1\ 



(1 - <7 r2 ^*)(l - g™- 1 -!-*) 



= ^ri,r 2 / -4-lxX (g * ) 

Now the constant already fixed so that the functions p r s 1,T2 are completely 

determined by property (d). Taking u — g~*~ s / 2 in the first recurrence in (|10|l . we 
obtain the following relation 

(13) P?- r2 {q- t - s/2 )= P 7^{q- t - [S - 1),2 )x + {q- S+1 - t )- 
Property (e) means that 

p ri,r- (g -t-./2j = for teZ\I n . 2s (r 1 -s,r 2 -s). 
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1 (q 



-t- s /2 ) = ip r u r 2(t) 



This follows easily from the recurrence (|13|l . Indeed, we just need to observe which 
are the zeros of \ + an d apply (b), we leave the details to the reader. Therefore, it 
remains to see (f). Let us consider the functions 

n — 2s 

. t, r 2 — s — t, r\ — r 2 + 1, n — r% — s — t. 

It is not difficult to check that each function 7j 1,r " 2 arise from 7^i'[ 2 and (|13|l . 
Therefore, property (f) follows from (c) and a simple induction argument. □ 

Remark 3.12. In Theorem ETUI we have chosen the appropriate solutions of the 
functional equation (PJJ) and the recurrences for our further purposes. This can 
be justified by the following combinatorial meaning of these functions. First, by 
identity (0, we have 

^(g-*)g-« = \{( Xl ,X 2 ) GX ri xX r2 \ d(x 2 ,X 1 ) = t} | . 



In particular, gives 
(14) 



tGZ 



! (<rV 



t _ 


" n ' 




' n ' 




ri _ 


Q 


-T2- 



Second, if X™ 2s denotes the set of r-dimensional of an (n — 2s)-dimensional vector 
space £l s over K, we also have 

p?' r '(q- t — /2 )q- t =S r s 1 ' r * 



rn — 2s 



{(x lll2 )ex^xx;_. 

with Sl 1 ^ 2 = q siri - r2 ^ 1/2 Hq n - 1 q~ 1 ) 2 s- In particular, 

(15) ^{q-^q- 1 = q-' /2 Y,P?' r 'to~ t ~' /2 )'!~ t 

tGZ 

q ^- r ^)(q^q- l ) 2s 
{q r \q n - r \q r2 ,q n - r2 ] 



d{x 2 ,xi) = t\ 



n — 2s' 




' n — 2s' 


. T\ — S _ 


1 


r 2 — s _ 



7 s(r 2 - 



1 + 1) Cn" 



(q n ;q- r ) 



2 s 



' n ' 




' n ' 




Q 


.r 2 . 



Remark 3.13. Our choice in Theorem 13 . 1 II has also the following interpretation. 
The operator C ri ^ 2 is clearly self-adjoint with respect to the Hilbert-Schmidt inner 
product on HomQ(V ri , V r2 ). In particular, this property can be rewritten in terms 
of the operator L rijI . 2 via the mapping $ : PN(n,r 2 ) Home (V ri , V r2 ) . Then, it 
can be checked that the hypergeometric operator L rij? - 2 becomes self-adjoint with 
respect to the inner product 



(/,<?> 



E 

tein(n,r a ) 



The role of the factor q * in this expression will become clear in Lemma FOl below. 
The reader is referred to Section 3.3 of |5] for more on these orthogonality relations. 

4. The product formula 

In this section we study a product formula for the operators A^ 1 ' 1 " 2 in B. To that 
aim, our first task is to normalize these operators since there are only determined 
up to a constant factor. We choose the normalization provided by 

/ s ri ' r2 (l) =A^' r2 (0) = 1. 
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Following Remark 13.71 we have 



(16) 



d, 



tri,r 2 _ I i\s s(r 2 -ri) + (|) V± ' > 



(9""'" 1 ,? r2 ;?- 1 ) s 



In particular, Rodrigues formula ( R s (ri, 7-2) I becomes 
P 



(17) / 9 ri ' r2 ( M )/; i ' r2 (u) = (-i)v Cr9-ri) ~K s ) — « — ^ 



Since the representations of G into the spaces V ri . Sl and V r2 . S2 are equivalent if 
and only if s\ = s 2 , we clearly have 



A r3 ' r4 o A ri ' r2 = 

s 2 si " 



unless si = s 2 and r 2 = 7*3. In particular, in order to give an explicit formula for the 
product of two operators in Endo(V), it suffices to study the products A^ 2 ' 7 " 3 oA^ 1 ' 1 " 2 . 
In the following result, we assume by convention that 



n 
-1 







Theorem 4.1. If < ri, r 2 , r 3 < n and < s < N(r 1? r 2 ) A N(r 2 , r 3 ), we /icrae 
(P s (r 1 ,r 2 ,r 3 )) 



A 



r2,r 3 Q ^ri,r 2 



r-2 J 9 



-a: 



The proof of Theorem 14. II requires two preliminary lemmas. In the first one we 
reduce the proof of the formula I P s (?"i, f 2 , ^3) I to two particular cases. 



Lemma 4.2. The product formula ( P s (ri, r 2 



> r 3, 



is implied by: 



• ( P s (ri,r 2 ,r 3 ) \ for n = r 3 . 

• ( Ps(ri,r 2 ,r 3 ) I /or n < r 2 < r 3 . 



Proof. In what follows we shall write 

k(r, s) = 



.8-1. 



"71" 




n 


. S _ 


1 


. s - 1 . 



Let P(r, s) : V r — > V r , s be the orthogonal projection. Then, @ and |JB} give 
tr(P(r, a)) = dimV r>s = 

tr(A^) = ^AJ^,«)) = 
In particular, it is clear that 



l£X r 



P(r,s) 



ri q 



Assuming ( P s (n, r 2 , 7* 3 ) I for n =7*3, we claim that 

(18) P s (ri,r 3 ,r 2 ) <S> P s (ri,r 2 ,r 3 ) ^ P s (r 2 ,ri,r 3 ). 
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For instance, we have 

A ^3,r 2 A ri,rs = k ^ s )-^ ^,r 2 Q A ^r 3 Q ^ri.n 

= k(r 3 , S )k(r 2 , S )- 1 A^^oA^-'' 2 
= k(r 3 ,s)P(r 2 ,s)oA^ 
= k(r 3 ,s)A^. 

This proves P(ri , r 2 , r 3 ) P(t"i, r 3 ,r 2 ) under the assumption of P(?~ 2 , ^3, r 2 ). The 
other implications in (|18J) can be checked in a similar way. On the other hand, 
since that the transpositions (a, b, c) 1— > (a, c, 6) and (a, b, c) 1— > (6, a, c) generate all 
permutations of (a, 6, c), it suffices to prove | P s (r\, r 2 , rg) I in the particular case 
?'i < ?*2 < ^3- Therefore, we need only to assume the two cases stated above. □ 

Lemma 4.3. Let /, g G Ai(C*) so that f is regular at q~ l and g is regular at 



<1 



-t-1/2 



for all integer t. Assume also that one of the sets 



{t ez| f(q- 1 ) ^0} or {t e z I .g(-r*- 1/2 ) ^ 0} 

is finite. Then we have the following summation by parts formula 



Proof. We have 



tGZ 



g-l/2_ ? l/2 



-1/2 



E /(?"*) (5(«- t_1/a 

(/(g-H-Va 



D/(g-*)ff(g-*)?- 



s(T t+1/2 )) 



□ 



Proof of ( P s (ri, r 2 , I for ri = 7-3. By Schur lemma, the maps A r s 2 ' ri o AJ 1,r2 
and k(r 2 , s)A r s 1 ' Tl are proportional. Hence, it suffices to prove that both operators 
have the same trace. Arguing as in Lemma f4. 21 the operator A r s 1,Tl has trace 



n 
nig 



fn particular, we need to prove that 

tr(A; 2 - ri o Al 1 ^ 2 ) 



n 

.3-1. 



Let f s be a polynomial of degree s 

f s (u) = d s f s u s + ... 

where the dots stand for terms of lower degree. The action of D on the main 
coefficient of f s is given by 

«/2 _ a -s/2 
Bfs(u)^dJ s q J U^ + ..., 
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see Section 2.2 of jH]- The iteration of this formula leads to 

'J' l > q {q-l) s 

„7l— S+l n S . n —X 



(19) 



a,. 



s(r2 - ri)+ |(l) (g" s+1 ,g s ;g 1 ) s 

{q-l) s {q n - r \q r *\q- x )s 



where the last identity follows from l|16(l . Now we are ready to compute the trace 
of Ag 2,ri o A^ 1 ' 1 " 2 . We begin by writing this trace in terms of the eigenfunction 
/J 1 '*" 2 and the Rodrigues function p ri ' T2 . To that aim, we recall the combinatorial 
meaning of this function, see Remark 13.121 Then, we use the Rodrigues formula 
given in 117fl after the normalization of the functions /J 1,r2 . 

tr(A^oA^ 2 ) = K^idi^xiWs^idixu^)) 

= x>™(rv t /r , ^(?~ t )/r* ri (9 ri ~ ri ~ t ) 



s(r 2 -n)-i(^) , 



r— iv 

( q n-r ljq r 2 . q -l) s 
^ /? ,r 1(f -r 1 -t )D8tf ,r 2(ft)fii 



Now recall that 



is a polynomial of degree less that s in u = q~ l . In particular, Lemma 14.31 gives 



Then, summation by parts and formulas (|15|l and 119|) give 



tr(A^' ri o A^ 2 ) = 



2s(r 2 -n)-i(=)/ i\2 



(9"- r S^ 2 ;g- 1 ) 



_.(9 n - +1 ,9 , ;9- 1 ) 



" n ' 




" n ' 


-n _ 


9 


■ T2- 



' n ' 




" n ' 




' n~ 




n 




-r\ _ 


9 




A 


. s . 


1 


.8-1. 


) 



Notice that, in the use of Ijl9|l . we interchange the roles of r\ and r%. □ 

Proof of ( P s (ri, t-2, r^)\ for r\ < < r^. Given < n < r2 < n, we define the 
Radon transform ' 2 : V ri — > V r2 as follows 

IlCl2 
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The kernel of this operator preserves d so that lZ r ^ ' T2 £ Home ( V ri , V r2 ) . On the 
other hand, given (xi,X3) £ X ri x X r2 , identity © gives 

|x 2 e x r2 1 x 1 c x 2 c xA 

In particular, 





r 3 


- n ~ 




"?V 




r 2 ~ 








Si 


- ri _ 


9 


T2- 


q 


.n. 


y 


.ri. 



(20) 





r r 3 




T2 1 




l r 2 


U 


ri J 9 




I 










n 


1. 



N(n,r 3 ) 

H™ = £ w.(n,r a )A^ 



Since "fc^- 1 ' r2 G Home (V ri , V r2 ) , we decompose it as 
(21) 

s=0 

To calculate the coefficients w s (r 1; r 2 ), we observe from © 
tr(A?^oft£**) = K 2 ' ri {d{xi,X2)) = 

(xi,x 2 ): xi<Zx 2 

On the other hand, 



" n ' 




r 2 " 


-r 2 . 


9 


.n. 



C 



w s (ri,r 2 )A^^oA^» 



Therefore, applying P s (ri,r 2 ,ri) we obtain 

tr(A^o7^' r2 )=w s (ri,r 2 ) 



n ] r n 
- ri J 9 I r 2 J 9 



.a - 1 - 



This result leads to the exact value of w s (ri,r 2 ). Then, we can rewrite l12()[| using 
identity Q21[l. Although we leave the details to the reader, it is not difficult to check 
that this gives 



N(n,r 2 ) 

E 



n 
s - 1 



, A r 2 ,r 3 oA n,r 2 



N(n,r 2 ) 

£ ( 



- r2 J q 



s=0 



' n~ 




n 




. s _ 


i 


.8 - 1_ 





a: 



Now, since these operators are mutually orthogonal, we identify coefficients. □ 
Corollary 4.4. The Hilbert- Schmidt norm of A r s 1,T2 is 



Al- 



ms 







[;; 





1/2 



Proof. By Schur lemma, there exists some constant c s (ri,r 2 ) with 

(A^)*=c.(n,ra)A^. 
If we write this relation in terms of the kernels, 

X^' r2 (d(x 2 ,x 1 )) = c s (n,r 2 ) X r /' ri {d(x 1 ,x 2 )) for all ( Xl , x 2 ) £ X ri xX r2 . 
This is equivalent to 
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Identifying the main coefficients via (|f fijl . we obtain the value of c s (ri, r^) 
(\ri,r2\* _ Q s(ra— n) W ; ^ ; " / s a ra.-ri 

(^ ri ,? r2 ;g- 1 ) s s 

The given expression for the Hilbert-Schmidt norm arises from Theorem 14.11 □ 
Remark 4.5. Our choice of the basis B follows from the condition 

/ s ri ' r2 (l) = X r s ur2 (0) = 1. 

This normalization is very natural since in this way the kernels X^.' r become the 
spherical functions associated to the symmetric space X r . This will be an essential 
observation in Section [5] However, there exist other natural normalizations for 
A^ 1 ' 1 " 2 . For instance, Corollary 14.41 provides a normalization for which the basis B 
becomes orthonormal with respect to the Hilbert-Schmidt inner product. Moreover, 
combining the results obtained so far it is not difficult to provide the normalization 
for which the basis B is made up of unitary operators. 

5. An alternative proof for the product formula 



In this section we provide an alternative proof of I P s (ri, r2,r^)\ which does not 
use any tool from the theory of classical hypergeometric polynomials. In contrast, 
the main tools will be the characterization of spherical functions given in Theorem 
1 1.31 and the Radon transforms 

defined for < r\ < T2 < n. Along the proof, we shall obtain some identities 
for the kernels As 1 ' 1 " 2 which might be of independent interest. Before starting the 
proof, we recall that Lemma 14.21 does not use any argument from the theory of 
classical hypergeometric polynomials. In particular, we again reduce the proof of 
the product formula to the proof of those particular cases. 

Remark 5.1. Along the proof, we shall assume by convention that 
r tl 

i) =0 for any integer r not satisfying < r < n. 



r 



ii) Ag 1 ' 1 * 2 = for any integer s not satisfying < s < N(r*i, r2). 

5.1. Combinatorial identities. Let us consider a subspace x of Vt of codimension 
9(f2, a;) = t and let us fix an integer < k < n. Our first aim is to calculate the 
number of r-dimensional subspaces x r G X r of Q satisfying d(x r ,x) — k. Clearly, 
this parameter is invariant under the action of G. In particular, it depends on the 
codimension of x but not on x itself. Hence we define 

M(n, r, t, k) = |x r E X r | d(x r ,x) = fc| 
Lemma 5.2. We have 

M(n,r,t,k) = q k{n 



t-r+k) 


f 




' n — t~ 




.k_ 




r — k. 



Proof. Let 



A = 



w) e x r _ fc x x k | z c x, w n x = {0} j. 



Then, we compute the cardinality of A in two different ways. First we notice that an 
element x r € X r satisfies d{x r , x) = k if and only if it can be written as x r = z © w 
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" n — 1~ 




f 


.r — k _ 


i 


.k. 



with (z, w) 6 A. We have only one possible choice for z = x r fl x while w is any 
fc-dimensional subspace with x r = z (Bw. Then, it follows from (@J that 

|A| = M{ji,r,t,k)q k{r - k l 

On the other hand, we can count first how many possible z's can we plug in A by 
applying J3J) with (n — t, r — k) instead of (n, r). Then we need to count how many 
u>'s can we plug in A. To that aim we notice that, for any such w the element 
y = x © w is an (n — t + fc)-dimensional subspace containing x. The number of 
possible y's is given by Finally, we need to count how many w's do we have for 
a fixed y, which is given again by (@J). In summary, we find that 

q k{n-t)_ 

1 

Combining the expressions obtained so far, we obtain the desired result. □ 

In the following result we use Radon transforms and our formula for M(n, r, t, k) 
to obtain some useful relations between the kernels A^ 1 ,V2 corresponding to a fixed 
value of the parameter s. 

Lemma 5.3. Let < ri,r 2 ,r 3 < n and < s < n/2. Then, there exist absolute 
constants Co, Ci, C2, C3 and C4 such that for any t £ I„(ri, r 3 ), we have: 

(a) X r /' r3 (t)=C X r /^(r 1 -r 3 + t). 

(b) Ifr 2 <r 3 , 

Cl A^ 1 < r ° (t) = ]T M (r 3 , r 2 , t , k) AI 1 ^ (k) . 

k 

(c) ifn < r 2 , 

c 2 , A? ' r3 (t) = ^2 M(n - ri,n - r 2 ,t,k) \ r s 2 ' r3 (k). 



(d) Ifr 2 <n, 



c 3 A^ 1 < r ° (t) = V M (n , r 2 , n - r 3 + t , r 2 - r 3 + k) A? ^ (k). 



(e) Ifr 3 < r 2 , 



k 



c 4 A^ 1 < r3 (t) - ^ M (n - r 3 , n - r 2 , n - r 3 + t , n - r 2 + fc) Ag 1,r2 (fc). 
fc 

Proof. By Schur lemma, (A^ 1,r3 )* = CoAg 3 ' Tl for some constant Co independent 
of the variable t. Then, (a) follows from the relation between the corresponding 
kernels and the identity 

d(xi,x 3 ) = n - r 3 + d(x 3 ,xi). 
To prove (b) we write (again by Schur lemma) 

Hence, if (£1,2:3) 6 X n x X r3 

ciA^ 3 (5(2:3, xi)) - J2 K ur2 (d(x 2 ,x 1 )) 
12C13 



]T \ E X r2 I x 2 C s 3l 9(x 2 ,xi) = fc}| A^ r2 (fc) 
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Then we observe that 

M(r 3 ,r2,d(x 3 ,xi),k) = | ^x 2 G X r2 | a; 2 C x 3 , d(x2, x\ n £3) = fc|| 

= ||a;2 G X r2 I £2 C #3, 5(a;2,a;i) = fc||. 

To prove (c) we write C2 K r s 1,T3 — A£ 2 ' r3 o TZ^' r2 for some absolute constant C2. 
Proceeding as above, this gives 

c 2 X^{d(x 3 ,xi)) = ]T I {^2 G X r2 I Xl C x 2 , a(x 3 ,a; 2 ) - fc}| A^' rs (fc). 
fc 

To calculate the coefficient, we work in the dual space ft*. Let X stand for the 
set of linear subspaces of 0* and X r the subset of r-dimensional subspaces of ft*. 
Then, if x G X denotes the annihilator of a subspace x G X, we have 

d{x L ,z 1 -) = d(x ± )-d(x- L f)z ± ) = d{x^) - 8{{x + z)^) = 8{x + z) -d{x) = d(z, x). 

In particular, 

M(n - n,n - r 2 , d(x 3 ,xi), k) 

= I G X n _ r2 J ^2 C I; , ^(2-2 i x i ^ "^3 

= \{ X % ^ ^n-r 2 I x 2 C ^Tj ^( a; 2~5 a '3~) = 

= ||a;2 G X r2 J xi G x 2 , d{x 3 ,x 2 ) = 
Finally, (d) follows from (a) and (b) while (e) follows from (a) and (c). □ 

Theorem 5.4. The kernels of the operators in the basis B satisfy: 

(a) IfO<t<s and s < r < n — s, 

( n s—r—l. „ — 1 \ 

K''(t) = ~, Li TT ii ^' S (0) ; 

• V I (q"-r ;q -l) t s y " 

f.r+8-n-l..-h 

In particular, we have A£' s (0) 7^ and A"~ s ' r (0) 7^ for all s < r < n — s. 

(b) IfO < s < N(n,r 2 ), there exists a polynomial /J 1 ' 1 " 2 of degree < s uniquely 
determined by the condition 

f?' r2 (q- t ) = X?^(t) for tGl„(r 1 ,r 2 ). 

Moreover, the degree of fl 1,T2 is s and there are non-zero constants C\ and 
c 2 such that 

f-i,r 2 / x \^ (q r2 u,q 1 ) s _fc(u 1 ;g 1 )kU k k(r . 2 _ s+k) s , , 

is {u) ~ Cl 2- {qS - k . q -i )s _ k{qk; q -i )k « x * W. 



fc=0 



fro («""*; 9 _1 )-fc(9 fc ;9 _1 )fc 

( c ) IfO<s< N(ri,r 2 ), i/iere ermfc a non-zero absolute constant Co swc/i that 

r s ^(u) = c /; 2 ' ri (9 r2 - ri ^)- 
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(d) J/0 < s < N(n,r 2 ), we have / s ri ' r2 (l) ^ and 



/ s ri ' r2 (i) Ar s ' ra (o) 

Jn particular, we have 

fr i,r 2 ( n ri-r 2 \ („ri n n—r 2 .„—l\ 

Js [q ) _ „s(r 2 -n) w >_q 



{q r *\q- x )s 



(e) //0<s<N(n,r 2 ), 



(A 



mT 2 \ 
ri,r2\* Ja__vjv j(rj-fi) 

/ £f 2 ,r\ f-t\ q 



- A. r2 ' ri 



Proof. By Lemma 15. 21 and Lemma f5. 31 (e) with (ri,r 2 ,r3) = (r,s,s — 1), we have 



^ " ' 1 q r - s+t+1 X r /(t + l)+ q 



r - s + t + 1 
r — s + fc 





' n 


— r 


-r 


9 


.n 


— r 


- fc. 



9-1 



9-1 

- E[' 

= y^M(n - s + l,n - s,r - s + 1 + l,r - s + k) A!? s (fc) 

= c 4 AI> s - 1 ^) = 0. 
The first identity in (a) follows by solving the recurrence 

^s—r—t — l 



1 - q s 



1 



The second identity in (a) follows similarly from Lemma 15.21 and Lemma 15.31 (d) 
with (ri, r 2 , r^) = (n — s + 1, n — s, r). For the first identity in (b), we use Lemma 
15.21 and Lemma 15.31 (b) with (ri,s, r 2 ) instead of (ri,r 2 ,r3) 



^M(r 2 , S ,t,fc) A^ s (fc) 



rn-t 
s — k 



q k(r 2 -t-s+k) A^ s (fc) 



- mi 

k H 

\ - (g r2 ~ t ;g~ 1 )«-fc(g t ;g~ 1 )fcg~ tfc Jfcfo-s+fc^n,,,^ 
^ (g ,9 )s-k{q k ;q ^k 

Clearly, the right hand side is a polynomial in the variable u = q~ l of degree < s. 
This gives the first identity in (b) and proves the existence of such a polynomial. 
Uniqueness follows from |I„(ri, r 2 )| = N(fi, r 2 ) + 1 > s. The second identity in (b) 
follows in a similar way from Lemma l5.2l and Lemma l5. 31 (c) by taking (ri, n — s,r 2 ) 
instead of (fl, r 2 , rs). From (a) and the first identity in (b), we can write the main 
coefficient of /J 1 ' 1 * 2 as 



E 

k=0 



(_iy-k q r 2 k-( s 2 k ) 
(q s - k ;q~ 1 ) s _ k (q k ;q- 1 



( n s— n— 1, 

fe(r 2 -s+fe) W ; 



9" 1 )* 



(9™- 



A?' s (0). 
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Therefore, since the coefficients of A^ 1,s (0) in this sum are all positive (notice that 
(q s ~ ri ~ 1 ; q^ 1 )k is positive for all k > since s < r±), we deduce that the main 
coefficient does not vanish so that /J 1 ' 1 " 2 has degree s. This concludes the proof of 
(b). Property (c) is another way to write Lemma l53l fa). The property /J 1,r2 (1) ^ 
follows trivially from (a). The first identity in (d) follows by evaluating the first 
identity in (b) at u = q~ r2 and u = 1 and then applying (a). Similarly, for the 
second identity in (d), we evaluate the second identity in (b) at u = q ri ~ n and 
u = 1 followed by (a). In both identities, the transformation formula 

(z i q- 1 ) a =(-l)'q-®z'(g'- 1 z- 1 ;q- 1 ) B 

is needed. The last identity in (d) follows from the previous ones and (c). To 
prove the identity in (e), it suffices to notice that the constant Co in (c) is given by 
/; i,r2 (l)//I 2 ' ri (y" 2 ~ ri ) and apply (d). This completes the proof. □ 

5.2. Proof of the cases r\ < r 2 < 7-3 and n > r 2 > r 3 . We have already seen 
that /J 1 '*" 2 satisfies the condition /J 1:r2 (l) ^ for any < s < N(ri,r2). Hence, 
from now on we normalize the operators A r s 1,T2 in B requiring /J 1,r2 (l) = 1- In 
particular, now Theorem l5.4l (c) has the form 

(22) (A^ 2 )* = d }^ 1,S \ A?' ri with d(r,s) - " sr(9 ' 1 1)s 



Let us consider < n, r 2 , r 3 < n with n < r 2 < r 3 . Then we claim that 

P(r3,r 2 ,ri) =*> P(n,r 2 ,r 3 ). 
Indeed, let us use the same notation as in Section 0] 



Then (J22J) gives 

A r 2 ,r 3 Q A n,r 2 



k(r, s) = 



d(r 3 ,s 



1 

r Jq 



If 

s - 1 



d(r 2 ,s 
d(r 3 ,s 



d(n,s 
d(r 3 ,s 



(a?'-)*^4(a: 

d(ri,s) 
(A^' ri o A^ 3 ' r2 )* 



k(r 2 , S ) (A^^)* = k(r 2 , S )AI 



d(n,s) 

In summary, it suffices to prove the case r\ > r 2 > r 3 . Notice that this is clear since 
Lemma |4.2I obviously holds with r\ > r 2 > r 3 instead of r\ < r 2 < r 3 . However, 
we have proved the implication P(r 3 , r 2 , ri) =4> P(n, r 2j r 3) since we shall need both 
results in Paragraph s. 31 

Remark 5.5. Let us write again P(r, s) : V,- — > V riS for the orthogonal projection 
from V r onto V r , s . Then, using that Ap r (0) = /J' r (l) = 1 and arguing as in the 
proof of Lemma 14. 21 we have 

(23) P(r, S ) = AY = k(r, s)" 1 A^. 

In particular, it turns out that 

A^ 2 oA^ = k(r llS )A^ 2 , 
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That is, I P s { r i: r 2 ; r 3 ) I holds with r\ — r 2 or r 2 = r 3 . Moreover, recalling the 
definition of spherical function given in Section ^ and that X r is a finite symmetric 
space for any < r < n, we observe from (|23|l that the set of spherical functions 
associated to X r is 

U r ' r \0<s<rA(n- 



(n-r)j. 



Following Remark l5.5l we are now allowed to use the characterization of spherical 
functions provided by Theorem 1 1.31 That is, given y £ X r for some < r < n, we 
consider the isotropy subgroup of y 

G y = £ G | gy = yj. 

Then, Theorem II .31 gives 

(24) r±- K' r (d(9X, z)) = \r(d( X) y)) X r s ' r (d(y, z)), 

for any x,y,z £ X r . On the other hand, by Remark 15. 51 we have 

x:^(d( X2 , xl )) = J2 x r s ^(d(x 2l z 1 ))\:^(d( Zl ,x 1 )), 

A^a^O) = rr^ ]T A™ (a(z 2 ,s 2 )) A™ (0(z 2 , a*)). 
Combining these identities with H24[). we obtain 

£ A^^ca^,^)) = A: i ^(a(x 2 , yi ))A^^(a( 2/1 , a;i )), 

^ \ r /> r *(d(gx 2 , Xl )) = X r s ^(d(x 2 ,y 2 ))X r s ^(d(y 2 ,x 1 )). 

with (yi,y 2 ) £ X n x X r2 . Now, let us assume that r\ > r 2 > r 3 and let us take 
Xk £ X rfc for k= 1,2,3 with 13CX2C11. Clearly, for any g £ G^ we will have 
X3 £ x 2 £ so that 5(0:3,3x1) = 0. On the other hand, by Schur lemma we have 

A?**oA?^=k,(ri s r 2 ,r 3 )A^ 

for some constant k s (ri, r 2 , rs). Putting the previous results all together, we have 

k s (ri,r 2l r 3 ) = ^ rz ; r3 ) £ K ur3 (d(x 3 ,g Xl )) 

= i^E E x^{d{ x ^z 2 ))x:^{d{z 2 , gxi )) 

= u^E E \7^(d(x 3 ,z 2 ))X r s ^(d(gz 2 ,x 1 )) 

= J2 A?^(a(^,^))A?^(a(z 2 ,x 2 ))A: i ^(9(x 2 ,a: 1 )) 
Z2ex r2 

= k(r 2 , a) \?^{d(x Zl x 2 )) X7' r Hd(x 2 , Xl )) = k(r 2 ,s). 
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5.3. Proof of the case n = r^. In this paragraph we shall need to use another 
type of Radon transforms. Given an integer < r < n, we consider the Radon 
transform W c : V,- — > V„_ r defined as follows 

Tl r c (p(z) = 2J <P{ X ) f° r ^ S V n -r- 

x: xPtz={0} 

Since the kernel of VJ C is invariant under the action of G, VJ C € HomG(V r , V n _ r ). 
Lemma 5.6. If n + ?'2 < n, we Ziaue 7?^ 2 o A^ 1 ' 1 " 2 = m(r2, s) h. r s 1 ' n ~ r2 with 

m(r, s) - (-ny^X"-^) (g "" r;g " 1)s . 

(<7 r ;<? l ) s 

Proof. By Schur lemma, we know the existence of a constant c such that 

When ri + r 2 < u, we can take xi £ X ri and Z2 € X„_ r2 such that x\ C 22- Then, 
d(z 2 , xi) = n — t*i — r2 and identity J3J gives 

cA^"- r2 (n-ri-r 2 ) = ]T A^ 2 (3(z 2 , a*)) 

3:2: aJ 2 nz2 = {0} 

= q r2{n ~ r2) \ r s 1 ' r2 {r 2 ). 

Since we have 

X r s 1 ' n - r2 (n-r 1 -r 2 ) = /^.«-^( g ^+^-"), 
the desired relation can be easily checked by applying Theorem l5.4l (d). □ 

Now we are ready to complete the proof. Arguing as in Section 0] the product 
formula 

(25) A^ ri o A^' r2 = k(ra, a) A r s uri 

is equivalent to 

tr(A^ 2 - ri o Ap< r2 ) = 



ri iqlr 2 iq 



Since tr(Ap ,ri o A r s 1,r2 ) = t^A^ 1 ''' 2 o A^. 2 ' ri ), we assume without lost of generality 
that n < r 2 . In particular, we have s < 7*1 < r2 < n — s. Multiplying on the 
left (resp. right) of (|25|l by A r s 2 ' s (resp. A^ ri ) and applying the results obtained in 
Paragraph l5.2l it turns out that the proof of (f^ is equivalent to the proof of 

A^ s oAr 2 =k(r 2 , S )Ar. 

Now, multiplying on the left by 1Z S C and applying Lemma l5.6l (notice that r 2 + s < n 
and s + s < n), the proof of (|25l) becomes equivalent to 

A r 2 ,„- SoA s, r2 = k ( r2jS ) A «,n- S . 

However, this identity holds since s < r 2 < n — s. Therefore, the proof is completed. 
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